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Abstract 

In |W2] West conjectured that there are 2(3n)!/((n+l)!(2n+l)!) two-stack sortable 



I permutations on n letters. This conjecture was proved analytically by Zeilberger in ||Z[ 



' Later, Dulucq, Gire, and Guibert |DGG] gave a combinatorial proof of this conjecture 



r"| ■ In the present paper we study generating functions for the number of two-stack sortable 

"j^ i permutations on n letters avoiding (or containing exactly once) 132 and avoiding (or 

^ I containing exactly once) an arbitrary permutation r on fe letters. In several interesting 

cases this generating function can be expressed in terms of the generating function for 
the Fibonacci numbers or the generating function for the Pell numbers. 
X ■ 

^ • Keywords: Two-stack sortable permutation; restricted permutation; pattern-avoiding 

permutation; forbidden subsequence; Fibonacci number; Pell number 



1 Introduction and Notation 

Let Sn denote the set of permutations of {1, ... , n}, written in one-line notation, and suppose 
n E Sn and a E Sk- We say tt avoids a whenever vr contains no subsequence with all of the 
same pairwise comparisons as a. For example, the permutation 214538769 avoids 312 and 
2413, but it has 2586 as a subsequence so it does not avoid 1243. If vr avoids a then 
a is sometimes called a pattern or a forbidden subsequence and vr is sometimes called a 

*MR Subject Classification: 05A15 
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restricted permutation or a pattern- avoiding permutation. In this paper we will be interested 
in permutations which avoid several patterns, so for any set R of permutations we write 
Sn{R) to denote the elements of Sn which avoid every element of R. For any set R of 
permutations we take Sn{R) to be the empty set whenever < and we take Sq^R) to be 
the set containing only the empty permutation. When R = {tti, 7^2, . . . , tt^} we often write 

Sn{R) = Sn{7li, 7l2,..., VT^). 

Pattern avoidance has proved to be a useful language in a variety of seemingly unrelated 
problems, from singularities of Schubert varieties [ pS[] , to Chebyshev polynomials of the 
second kind [|CW| , 



Kr, MVl 



to rook polynomials for a rectangular board |[MV2|| . There 
is a particularly fruitful interplay between pattern-avoiding permutations and the study of 
various sorting algorithms. One example of this interplay occurs in |[W2| , [W3|| , where West 
studied permutations sortable by two passages through a stack, under the condition that no 
element of the permutation may be placed on top of a smaller element. Such permutations 
are now known as two-stack sortable permutations; we write P„ to denote the set of two-stack 
sortable permutations in Sn- West characterized two-stack sortable permutations in terms 
of pattern-avoidance, showing that Vn is the set of permutations in Sn which avoid 2341 and 
3241, except that the latter pattern is allowed whenever it is contained in a subsequence of 
type 35241 in the permutation. West |[W2| , |W3|| conjectured that for all n > 0, 



I'Pnl 



2(3n)! 



(n + l)!(2n + 1)!' 



This conjecture was first proved by Zeilberger in 0. Zeilberger's proof is analytical, rather 
than combinatorial. Later, Dulucq, Gire, and Guibert | |DGG| gave a combinatorial proof of 
this conjecture (see also |BT|, pUW| , |T[). 

In this paper we consider two-stack sortable permutations which also avoid additional 
permutations. Accordingly, for any set R of permutations, we write Vn{R) to denote the 
set of two-stack sortable permutations in Sn which avoid every pattern in R, and we write 
Pr{x) to denote the generating function given by 



PR{x)=J2\'Pn{R)\ 



X" 



n=0 



We will encounter numerous sequences in this paper, many of which will be closely related 
to two particular sequences: the sequence of Fibonacci numbers and the sequence of Pell 
numbers. We write Fq, Fi, . . . to denote the sequence of Fibonacci numbers, which are given 
by Fq = 0, Fi = 1, and F„ = F„_i + F„_2 for n > 2. We observe that the generating function 
for the Fibonacci numbers is given by 



X 



n=0 



1 — X 



X^ 



We also observe that F„ may be interpreted combinatorially as the number of tilings of a 
1 X {n — 1) rectangle with tiles of size 1x1 and 1x2. We write Po,pi, ... to denote the 
sequence of Pell numbers, which are given by po = 0, pi = 1, and Pn = 2pn-i + Pn-2 for 
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n > 2. We observe that the generating function for the Pell numbers is given by 

oo 

Ep-^"" = . o! (2) 



1 ~ 2x — X 

n=0 



We also observe that may be interpreted combinatorially as the number of tilings of a 
1 X [n — 1) rectangle with tiles of size 1x1 and 1x2, where each 1x1 tile can be red or 



blue. For more information on the Pell numbers, see @, |Be, pp. 122-125], @, and Q. 

In this paper we use generating function techniques to study those two-stack sortable 
permutations which avoid (or contain exactly once) 132 and which avoid (or contain ex- 
actly once) an arbitrary pattern in Sk- In Section ^we describe a method for enumerating 
'P„(132,r) for any r G P„(132). Using this method, we give several enumerations which 
involve the Fibonacci or Pell numbers. In Section § we use similar techniques to find gen- 
erating functions for the set of permutations in 7^^(132) which contain another pattern r 
exactly once. Again we give specific examples involving the Fibonacci or Pell numbers. In 
Section ^ we turn our attention to the set Q„ of permutations in Vn which contain exactly 
one subsequence of type 132. We give a technique for finding the generating function for 
those permutations in Q„ which avoid, or contain exactly once, a permutation r. Again we 
provide specific examples involving the Fibonacci or Pell numbers. We conclude the paper 
with examples of other problems which might be solved using similar techniques. 



2 Two-stack Sortable Permutations Which Avoid 132 
and Another Pattern 

In this section we describe a method for enumerating two-stack sortable permutations which 
avoid 132 and at most one additional pattern and we use our method to enumerate 'P„(132, r) 
for various r G Pfc(132). We observe that since 35241 contains a subsequence of type 132 
(namely, 354), for any set R of permutations we have 7^^(132, i?) = (132, 2341, 3241, i?). 
We begin with an observation concerning the structure of the permutations in ^^(132). 

Proposition 2.1 Fix n > 3 and suppose ir G 'P„(132). Then the following hold. 

(i) TT~^{n) = 1, 7i^^{n) = 2, or 7r'^{n) = n. 

(ii) The map from P„_i(132) to P„(132) given by 

IT n,iT 

is a bijection between P„-_i(132) and the set of permutations in P„(132) which begin 
with n. 

(iii) The map from P„_2(132) to P„(132) given by 

TT n — 1, n, vr 

is a bijection between P„_2(132) and the set of permutations in P„(132) whose second 
entry is n. 
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(iv) The map from Pn-i(132) to Pn(132) given by 



is a bijection between P„_i(132) and the set of permutations in P„(132) which end with 
n. 

Proof, (i) Suppose by way of contradiction that 2 < 7r~^(n) < n. Since vr avoids 132, the 
elements to the left of n in vr are all greater than every element to the right of n. Since 
there are at least two elements to the left of n and at least one element to the right of n, 
there must be a pattern of type 3241 or 2341 in vr in which n plays the role of the 4. This 
contradicts our assumption that vr G P„(132). 

(ii) Since the given map is clearly injective it is sufficient to show that vr G P„_i(132) if and 
only if TT G Pn(132). Since Pn(132) = 5^(132, 2341, 3241), it is clear that if vr G Pn(132) 
then vr G P„_i(132). To show the converse, suppose vr G P„_i(132). If n, vr contains a pattern 
of type 132 then the n cannot be involved in the pattern, since it is the largest element of 
n, TT. Then vr contains a pattern of type 132, a contradiction. By similar arguments for 2341 
and 3241 we find that n, vr G 'P„(132), as desired. 

(iii) ,(iv) These are similar to the proof of (ii). □ 

Theorem 2.2 For all n> 1, 

|P„(132)| =p„. (3) 



Proof. For notational convenience, we abbreviate P{x) = Pi32(x). By Proposition |2.1| (i), 
when n > 3 the elements of P„(132) may be partitioned into three classes: those which 
begin with n, those whose second entry is n, and those whose last entry is n. By Proposition 
p.l| (ii), the generating function for those elements which begin with n is x{P{x) — 1 — x). 
By Proposition pTl](iii), the generating function for those elements whose second entry is n 



is X {P{x) — 1). By Proposition p.l| (iv), the generating function for those elements which 



end with n is x{P{x) — 1 — x). Combine these observations to obtain 

P(x) = 1 + X + 2x^ + x(P(x) - 1 - x) + x'^{P{x) - 1) + x{P{x) -1-x). 

Solve this equation for P{x) and compare the result with (0) to complete the proof. □ 

We also give a combinatorial proof of @. 

Theorem 2.3 For all n > 1, there exists a constructive bijection between P„(132) and the 
set of tilings of a 1 x {n — 1) rectangle with tiles of size 1x1 and 1x2, where each 1x1 
tile can be red or blue. 

Proof. Suppose we are given such a tiling; we construct the corresponding permutation as 
follows. Proceed from right to left, placing one number in each box of a tile. If the rightmost 
empty tile is a 1 x 2 tile then fill it with the two smallest remaining numbers, in increasing 
order. If the rightmost empty tile is a red 1x1 tile then fill it with the largest remaining 
number. If the rightmost empty tile is a blue 1x1 tile then fill it with the smallest remaining 
number. When all tiles have been filled, one number will remain. Place this number in the 
leftmost position in the permutation. To obtain a permutation in P„(132), take the (group 
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theoretic) inverse of the permutation constructed by the process above. It is routine to 
construct the inverse of the map described above, and thus to verify it is a bijection. □ 



Though elementary, Proposition |2.1| enables us to easily find Pi32,t{x) for various r. For 
instance, we have the following result involving t = 12 . . . d. 

Theorem 2.4 (i) Pi32,i(a;) = 1. 
(ii) -Pi32,i2(a;) 



1 — X 
(iii) For all d >3, 

d-3 



;i - x) ^ (1 - X - x^Y+^x'^-^-'' + x'^-'^ 
•=o 

(1 — x)(l — X — x'^Y 



\32,12...d{x) - '-p- — -^-^3^ . (4) 



Proof, (i) Observe that only the empty permutation avoids 1. 

(ii) Observe that for all n > 0, the only permutation in Sn which avoids 12 is n,n — 



1,...,2,1. 



(iii) We argue by induction on d. By Proposition for all d > 3 we have 
Pi-i2,i2...d{x) = l+a;+2x^+x(Pi32,i2...d(a;)-l-x)+x^(Pi32,i2...d(x)-l)+x(Pi32,i2...d-i(a;)-l-x). 
Solve this equation for Pi32,i2...d(a^) to find that for all d > 3, 



X 

Pl32,l2...d{x) = 1 + ^Pl32,12...d-l(2;). (5) 

1 — X — X^ 



Set (i = 3 in (^, use (ii) to eliminate Pi32,i2(a^)5 and simplify the result to obtain 

(1 — x)(l — X — X^) + X 



Pl32,123(3;) 



1^1 — x)(l — X — X^ 



Therefore (^) holds for d = 3. Moreover, if (^ holds for d then it is routine using (|^) to 
verify that (||) holds for ci + 1. □ 

Corollary 2.5 For all n > 1, 

|P„(132, 123)1 = P„+2-l (6) 

and 

|P„(132, 1234)1 = 1 - |p„+i + (P„+3 + P„+i) . (7) 

5 5 

Proof. To prove (|^), first set d = 3 in @) and simplify the result to obtain 

1 X + 1 

Pl32,123(a;) = 1 - \- o- 

1 — X 1 — X — x^ 
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Compare this last line with (|l]) to obtain (|). To prove (0), first set c? = 4 in (||) and simplify 
the result to obtain 

1 2 1 

A32,i234(a;) = 1 + ^ + 



1 — X \ — X — {\ — X — x'^Y' 
It is routine to verify that 



(1 — X — x^)^ 

^ ' n=0 

and (0) follows. □ 

Next we consider Pi32,di2...(d-i)(a;). 
Theorem 2.6 (i) -Pi32,2i(2^' 



fii) ^'132,312(3; 



1 — X 

(1 — X — x^)(l — x) + x(l + x) 



(1-X)2 

(iii) For a// > 4, 

-pL32,dl2...(d-l)(a^) = 

d-4 

(1 - X)(l - X - X^)'^-^ + (1 _ a;2) ^ _ ^ _ ^2)r+l^d-3-r ^ ^d-2(^^ ^ ^) 

^1^5 /ON 

(l-x)2(l-X-x2)rf-3 • 

Proof. (i) Observe that for all n > 0, the only permutation in Sn which avoids 12 is 
1, 2, . . . , n — 1, n. 



(ii),(iii) By Proposition for all d > 3 we have 

Pl32,dl2...{d-l)(yX) 

= 1 + X + 2x^ + x(Pi32,12...d-l(a;) - 1 - x) + x2(Pi32,i2...d-i(a;) - 1) 

+xiPl32,dl2...{d-l){x) - 1 - X). 

Solve this equation for Pi32,di2...{d~i){x) and use Theorem |2.4|(ii),(iii) to eliminate the factor 
^'i32,i2...d-i(a;), obtaining (§). □ 

Corollary 2.7 We have 

|P„(132,312)| = 2n -2 in>2) (9) 

and 

|P„(132,4123)| = F„+4-2n-2 (n > 0). (10) 
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Proof. To prove (P), first observe from Tlieorem |2.6| (ii) tliat 

2 4 



^132.312 W = 3 + X + 



(1 — xy 1 — X 



Now (P) follows from the binomial theorem. The proof of ([l^) is similar to the proof of (|^). 

□ 

We conclude this section by describing a recursive method for computing Pi32,t{x) for 
any permutation r G Vk{lS2). Observe that this allows us to compute Pi32,r(a;) for any 
permutation r G Sk, since Pi32,T(a^) = -Pi32(a^) if t G 5"^ but r ^ 7^^(132). 
Theorem 2.8 Fix k > 3 and r G Pfc(132). Observe by Proposition ^[J that exactly one of 
the following holds. 

(a) There exists r' G Vk-i{lS2) such that r = /c, r'. 

There exists t' G Pfc_2(132) s'uc/i that t = k — 1, k, t' . 

(c) There exists t' G Pfc_i(132) snc/i that t = r', k. 

Then the following also hold. 

(i) If (a) holds then 

_ l-x-x^ + x{l + x)Pi32y{x) 

^132,t[X) — — . [i-i-) 



(ii) // (b) holds then 



(iii) If (c) holds then 



1 — X — 



(13) 



Proof. The proofs of (i)-(iii) are similar to the proofs of Theorem |2.4| (iii) and p.6| (iii). □ 
Corollary 2.9 

|P„(132, 3412) 1 = 3- 2"'2 _ i {n > 2) (14) 
|P„,(132, 45123)1 = 3-2""i -F„+3 + l {n > 1) (15) 

6 Tl -\- \ 

|P„(132, 561234)1 = 3 ■ 2" - 1 - - ^— (F„+4 + i^„+2) {n > 1) (16) 

Proof. To prove ([T^), first use Theorems |2]^(ii) and [^.4| (ii) to find that 

5 1 1 3 1 

i^i32,34i2(x) - 4 + 2^ - + 413^- 

Now ([I^) is immediate. The proofs of (|15D and (|T^) are similar to the proof of (0). □ 
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Proposition 2.10 For all d > 2, 

d-3 



d-2 



\Z24...21{X) = — 7- -TIZ] • (17) 



{i-xy-^ 

Proof. We argue by induction on d. When d = 2, line (|T^) reduces to Theorem ^]^(ii). If 
(p!7|) holds for a given d > 2 then it is routine using Theorem |2]^(i) to show that {\i7\} holds 
for d+1. a 

Corollary 2.11 We have 

|P„(132,321)| = 2n-2 (n > 2), (18) 
|P„(132,4321)| = 2r22-8n + ll {n>3), (19) 

and 

4 128 

|P„(132, 54321)1 = -n^ - 12^2 + —n - 52 (n > 4). (20) 

Proof. To prove ([T^), first set d = 3 in (|T^ and simplify the result to find 

4 2 

-Pl32,321 (x) = 3 + X - h 



1 ~ X (1 — x) 



Now (|TB|) follows from the binomial theorem. The proofs of ([TP|) and (^) are similar to the 
proof of (m). □ 



3 Two-stack Sortable Permutations Which Avoid 132 
and Contain Another Pattern 

Fix d > 1 and set 

P(132) = |JP„(132). 

n>0 

Inspired by results such as those found in | JBCS| | and [Q, we consider in this section the 
generating function for permutations in P(132) according to the number of patterns of type 
12 ... (i, (i 12 ... (i — 1, or (i c? — 1 ... 21 they contain. We begin by setting some notation. 

Definition 3.1 For any permutation r G 7^^(132) and any r > 1, let bn,r denote the number 
of permutations in P„(132) which contain exactly r subsequences of type r. Then we write 

oo 

B:{x) = Y,bn,rX''. (21) 

n=0 

We now consider the case in which r = 12 . . . c?. 

Definition 3.2 For any permutation vr and any d > 1, we write 12. . .d{iT) to denote the 
number of subsequences of type 12 ... c/ in it and we write |7r| to denote the length of it. 
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Theorem 3.3 Let xi, X2, ■ ■ ■ denote indeterminates. Then we have 



n ^i" 

E n-f *' = i + E^ v- 

.e^(i32)d>i n 1 - n - n x'^-^^xj^r- 

m=l \ j>l j>l 



Proof. For notational convenience, set 



A{x,,x2,...)= Yi n 

7re-p(132) d>l 



12...d(7r) 



By Proposition |2.1| , 

A(xi,X2,...) = 1 + xi + + + xi (A(xi,X2, . . .) - 1 - 

+ X1X2 (A(xi, X2, . . .) - I) + Xi (A(xiX2, X2X3, . . .) - 1 - X1X2) 



Solve this equation for yl(xi, X2, . . .) to obtain 

Z;iA(xiX2,X2. 

1 — Xi — X^X2 



XiA(xiX2,X2X3, . . .) 
A(Xi, X2, . . .) = IH ■ ^ • 



Iterate this recurrence relation to obtain fBl). □ 



For a given d > 1, we can use ( p2D to obtain the generating function for V{132) according 
to length and number of subsequences of type 12 ... 
Proposition 3.4 For all d > 3, 

(1 — xYyV — X — x^Y ^ 

Proof. In set xi = x, x^ = y, and Xj = 1 for i 7^ 1, d. Expand the resulting expressing 
in powers of y; we wish to find the coefficient of y. Observe that only the terms in the sum 
for which n = d— loTn = d contribute to this coefficient. These terms are 



(1 — X — x'^)^ ^(1 — X — x'^y) 

and 

x'^y 



(1 — X — x^)"* ^(1 — X — x^?/)(l — xy — x^y^) 
respectively, and p3|) follows. □ 



Corollary 3.5 For alln > 0, the number of permutations in P„(132) which contain exactly 
one subsequence of type 123 is Fn+2 — n — 1. 
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Proof. Set d = 3 in (|23|) and simplify the result to find 

a; + 1 1 



X — (1 — x)2 



Now the result follows from (|T]) and the binomial theorem. □ 

As another application of (P^, we now find the generating function for 7^(132) according 
to length and number of right to left maxima. To do this, we first find the number of right 
to left maxima in a given permutation tt in terms of 12 . . . c/(7r). 

Definition 3.6 For any vr G Sn, we write rmax{TT) to denote the number of right to left 
maxima in it. 

Proposition 3.7 For all tt G S'„(132), we have 

n 

rmax{7T) ='^{-iy+h2...d{7c). (24) 

d=l 

Proof. Fix vr G 5'„(132) and fix i, 1 < z < ra. We consider the contribution of those 
increasing subsequences of vr which begin at 7r(i) to the sum on the right side of (^4]). If 
7T{i) is a right to left maxima, then the only increasing subsequence which begins at 7r{i) has 
length one. Therefore each right to left maxima contributes one to the sum. If 7r{i) is not a 
right to left maxima then we observe that because vr avoids 132, the elements to the right of 
TT{i) which are larger than 7r{i) are in increasing order. Therefore the contribution of 71(2) to 
the sum is 

where k is the number of elements in vr larger than 7r(i) to the right of vr(i). Combine these 
observations to obtain (p^. □ 



Proposition 3.8 We have 



^\n\yrmax{7r) ^ ^ ^ ^Vi^^ ^ ^'^ ) ^25) 

^ (1 — xy — x'^y)(l — 2a; — x"^) 

7rg-p(132) \ y yj\ I 

Proof. In (p^), set X\ = xy and Xi = y^"^'''^^ for i >2. Use the facts 



i=0 

and 



5nl 



Ex" a;(l — X — X ) 

(1 - X - x2)"-i ~ 1 - 2x - x2 

n=l ^ ' 



to simplify the result. □ 
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Corollary 3.9 For all r > 1, let a.,.^„ denote the number of permutations in Pn(132) with 
exactly r right to left maxima. Then 

f:.,„.-^ -^'^^f7-r-^ (26) 

^-^ 1 — 2x — 

n=0 



In particular, 



and 



ai^n = Pn-i {n>2), (27) 

a2,n = Pn-2 + Pn-3 (^^ > 4), (28) 



a3,n = 2pn-3 in>Q). (29) 
Proof. To obtain ([26|), first observe that 

1 



1 — xy — x'^y 



^x"(x + l)V. 



Combine this with (|25| ) to find that 



(^•) ^ x(l — X — a;'"?/'" ' ^(i + X) 
1 — 2x — x"' 

7reT'(132) n=0 



Take the coefficient of y"^ in this last line to obtain (|26D. To obtain (pTl) , set r = 1 in (^ 
and compare the result with (^ . The proofs of ( PBD and ( |25| ) are similar to the proof of 

□ 



We remark that (pY| ) and (^) can also be obtained directly from Proposition 2A and 
Theorem |2.2| . For instance, if vr G P„(132) has exactly one right to left maxima then it must 
end in n. By Proposition |2ri|(iii) and Theorem p.2| , there are exactly Pn-i such permutations 
in P„(132), and ( pT)) follows. A similar but slightly more involved argument proves (p^). 

Next we find -Bdi2...d-i(3;)- 
Theorem 3.10 (i) 

3 

bU^) = ^30) 

(ii) For a// > 4, 

^ x^ 

(1 — X)'^(l — X — X^)'^ ^ 

Proof, (i) Fix vr G 'P„(132) such that vr contains exactly one subsequence of type 312. We 
consider the three cases of Proposition p.l| . If vr(l) = n then n > 3 and vr = n,n — 2,n — 
1, n — 3, n — 4, . . . , 2, 1. If vr(2) = n and vr(l) = n — 1 then vr does not contain exactly 
one subsequence of type 312. If vr = vr',n then vr' G P„_i(132) and vr' contains exactly one 
subsequence of type 312. Combine these observations to find 

Bli2ix) = j^^ + xBl,^{x). 
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Solve this equation for _B|^2(2^) obtain (|30D . 

(ii) Fix vr G P„,(132) such that vr contains exactly one subsequence of type dl2 . . .d — 1. 
We consider the three cases of Proposition If 7r(l) = n then 7r(n) = n — 1, since vr 
avoids 132, 2341, and contains exactly one subsequence of type dl2 . . . — 1. Moreover, it is 
routine to show that the map vr' h-> n, vr', n — 1 is a bijection between those permutations in 
P„_2(132) which contain exactly one subsequence of type 12 ... rf — 2 and those permutations 
in 'P„(132) which contain exactly one subsequence of type dl2 . . .d — 1. If 7r(2) = n and 
7r(l) = n — 1 then tt does not contain exactly one subsequence of type dl2 . . .d — 1. If 
7r(n) = n then vr = 7r',n, where vr' G P„_i(132) and vr' contains exactly one subsequence of 
type dl2 . . .d — 1. Combine these observations to find 



-^dl2...d-l(^) ~ ^^-^12...d-2(^) + ^-^dl2...(i-l 



X) 



Now use (l2^) to eliminate -Bi2...d-2(^) ^'^'^ solve for -B^i2. ci-i(^) obtain (|3TD . □ 

Corollary 3.11 For all n>\, the number of permutations in 'P„(132) which contain exactly 
one subsequence of type 51234 is 



n+2 



n + 1 
2 



Proof. Set (i = 5 in (|3l|) and simplify the result to find 



1 x + 1 3_ 2 



^51234V-; 1 _ x (1 - x)^ (l-x)3- 

Now the result follows from (|l]) and the binomial theorem. □ 

We conclude this section by finding ...21 (^)- 

Theorem 3.12 For all d> I, 



Proof. This is similar to the proof of Theorem 3.10. □ 



4 Two-stack Sortable Permutations Which Contain 132 
Exactly Once 

We now turn our attention to another set of permutations in Vn- 

Definition 4.1 For all n > 0, we write Qn to denote the set of permutations in Vn which 
contain exactly one subsequence of type 132. 

Using the methods of the previous section, one can obtain generating functions for those 
permutations in Q„ which avoid, or contain exactly once, any permutation r G Sk- In 
this section we illustrate these derivations with examples in which the resulting generating 
functions are given in terms of the generating functions for the Pell or Fibonacci numbers. 



We begin with an analogue of Proposition 
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Proposition 4.2 Fix n > 3 and suppose it & Qn- Then the following hold. 

(i) 7r(l) = n, 7r(2) = n, 7r(n) = n, or tt = 3142. 

(ii) The map from Qn-i to Qn given by 

71 n,7r 

is a bijection between and the set of permutations in Qn which begin with n. 

(ill) The map from Qn-2 to Qn given by 

7r I— s> n — 1, n, TT 

is a bijection between Qn-2 o-nd the set of permutations in Qn whose second entry is n 
and in which n does not take part in the subsequence of type 132. 

(iv) The map from Qn-i to Qn given by 

TT 7r,n 

is a bijection between Qn-i and the set of permutations in Qn which end with n. 

(v) The map from Vn-2 to Qn given by 

n i-^ n — 2,n, tt', 

in which ir' is the permutation of 1,2, ... ,n — 3,n — l which results from replacing n — 2 
with n — 1 in TT, is a bijection between Vn-2 <md the set of permutations in Qn whose 
second entry is n and in which n takes part in the subsequence of type 132. 

Proof, (i) Suppose 2 < TT~^{n) < n. We consider two cases: n takes part in the subsequence 
of type 132 or n does not take part in the subsequence of type 132. 

If n does not take part in the subsequence of type 132 then the elements to the left of n 
are all greater than every element to the right of n. Since there are at least two elements to 
the left of n and at least one element to the right of n, there must be a pattern of type 3241 or 
2341 in which n plays the role of the 4. This contradicts our assumption that vr G Q„ C P„. 

Now suppose the subsequence of type 132 in tt is a, n, h. Since vr contains no other 
subsequence of type 132, all elements to the left of n other than a are greater than h and all 
elements to the right of n other than h are less than a. Observe that if there are additional 
elements both to the right and left of n then one of these elements on each side of n, together 
with a and n, form a pattern of type 2341 or 3241. Therefore, there can only be additional 
elements on one side of n. Since we have assumed 2 < 7r~^(n) < n, there must be at least 
one additional element to the left of n and no additional elements to the right of n. If there 
are two (or more) additional elements to the left of n then they combine with n and h to 
form a 3241 or a 2341 pattern, so there must be exactly one additional element to the left 
of n. Now the only possibilities are vr = 3142 and tt = 1342. But 1342 has more than one 
subsequence of type 132, so we must have vr = 3142, as desired. 

(ii)-(v) These are similar to the proof of Proposition |2.1| (ii). □ 



Using Proposition we now find the generating function for Q„. 



13 



Theorem 4.3 We have 



Eig.k.- ^^';,?-rir^' . (33) 



n=0 

Moreover, 



\Qn\ = \ {51npn - U5pn - 2lnpn+i + 60pn+i) (ri > 3). (34) 



Proof. For notational convenience, set 

oo 

Q{x)=Y,\Qn 



n=0 



To obtain (pBf), observe that by Proposition [4.2| we have 

Q(s) = 2xQ{x) + a;2Q(x) + x2Pi32(x) + x\ 

Use (|^) and (0) to ehminate -Pi32(a;) and solve the resuhing equation for Q{x) to obtain 
To obtain (0), first observe that 

^, , , 87x-36 49X-51 

Q[x) = -x' + 4a; - 15 + ^ - -. 

[1 — 2x — x^Y 1 — 2x — x^ 

Now observe that 



oo ^ 

^-((n + lK + (3n + 4K+i) 



(l-2x-x^y ^4 

^ ^ n=0 

Combine these observations with (0) to obtain □ 



X . 



Proposition [4.2| enables us to find the generating function for those permutations in Q„ 
which avoid, or contain exactly once, any permutation r. We illustrate how this is done 
by considering those permutations in which r = 12 . . .d appears exactly once. This case 
is particularly interesting, since the result generating function is expressed in terms of the 
generating function for the Fibonacci numbers. 

Definition 4.4 For any permutation r G Sk, let Cn denote the number of permutations in 
Qn which contain exactly one subsequence of type r. We write D].{x) to denote the generating 
function given by 

oo 

Dl{x) = Y,CnX\ 
n=0 

Theorem 4.5 For all d > 2, 

1 / ^ _ {d- 2)2;'^+^ 

^''■■■'^"^^ ~ {l-xy{l-x-xY-'' ^^^^ 
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Proof. The case d = 2 is immediate, so we assume d > 2 and argue by induction on d. By 
Proposition [4.2|, we find 



Use ( P^D to eliminate -Bi2...d(a^) and solve the resulting equation for ^{x), obtaining 



Dl 



1 / x'^'^'^ 



a: — x^)' 



Use induction to eliminate ^-li-'^) dH) follows. □ 



Corollary 4.6 The number of permutations in Qn which contain exactly one subsequence 
of type 123 is 

n 2 

- {Fn+i + Fn-i) - - (5F„+i - Fn-i) +n + 2 (n > 4). 
Proof. This is similar to the proof of (|^). □ 



5 Directions for Future Work 

In this section we present several directions in which this work may be generalized, using 
similar techniques. 

1. In Section ^ we enumerated two-stack sortable permutations which avoid 132 and one 
additional pattern. Using the same techniques, one ought to be able to enumerate two- 
stack sortable permutations which avoid 132 and two or more additional patterns. For 
instance, for any (i > 2, it should be possible using our techniques to find the generating 
function -Pi32,i23...d,2i3...d(a;). Moreover, we expect that this generating function will be 
given in terms of the generating function for the Fibonacci numbers. 

2. In Section ^ we enumerated two-stack sortable permutations which contain exactly 
one subsequence of type 132 and avoid (or contain exactly once) another pattern. One 
ought to be able to use similar techniques to enumerate permutations which contain 
exactly r subsequences of type 132, for a given r. 

3. In Section |^ we found generating functions for two-stack sortable permutations with 
respect to various statistics on permutations. Using similar techniques, one ought to 
be able to find generating functions for two-stack sortable permutations with respect to 
additional statistics, such as rises, descents, right to left maxima, right to left minima, 
left to right maxima, and left to right minima. 
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